System and method for supporting work sharing muxing in a cluster

ABSTRACT

A system and method can provide efficient low-latency muxing between servers in the cluster. One such system can include a cluster of one or more high performance computing systems, each including one or more processors and a high performance memory. The cluster communicates over an InfiniBand network. The system can also include a middleware environment, executing on the cluster, which includes one or more application server instances. The system can include one or more selectors, wherein each said selector contains a queue of read-ready file descriptors. Furthermore, the system can include a shared queue, wherein the read-ready file descriptors in each said selector can be emptied into the shared queue. Additionally, a plurality of muxer threads operates to take work from said shared queue.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is related to the following patent applications, eachof which is hereby incorporated by reference in its entirety:

U.S. patent application Ser. No. 13/109,849, filed May 17, 2011,entitled “SYSTEM AND METHOD FOR ZERO BUFFER COPYING IN A MIDDLEWAREENVIRONMENT;

U.S. patent application Ser. No. 13/170,490, filed Jun. 28, 2011,entitled “SYSTEM AND METHOD FOR PROVIDING SCATTER/GATHER DATA PROCESSINGIN A MIDDLEWARE ENVIRONMENT”;

U.S. patent application Ser. No. 13/109,871, filed May 17, 2011,entitled “SYSTEM AND METHOD FOR PARALLEL MUXING BETWEEN SERVERS IN ACLUSTER”; and

U.S. patent application Ser. No. 13/167,636, filed Jun. 23, 2011,entitled “SYSTEM AND METHOD FOR SUPPORTING LAZY DESERIALIZATION OFSESSION INFORMATION IN A SERVER CLUSTER”.

COPYRIGHT NOTICE

A portion of the disclosure of this patent document contains materialwhich is subject to copyright protection. The copyright owner has noobjection to the facsimile reproduction by anyone of the patent documentor the patent disclosure, as it appears in the Patent and TrademarkOffice patent file or records, but otherwise reserves all copyrightrights whatsoever.

FIELD OF INVENTION

The present invention is generally related to computer systems andsoftware such as middleware, and, is particularly related to systems andmethods for muxing between servers in a cluster.

BACKGROUND

Within any large organization, over the span of many years theorganization often finds itself with a sprawling IT infrastructure thatencompasses a variety of different computer hardware, operating-systems,and application software. Although each individual component of suchinfrastructure might itself be well-engineered and well-maintained, whenattempts are made to interconnect such components, or to share commonresources, it is often a difficult administration task. In recent years,organizations have turned their attention to technologies such asvirtualization and centralized storage, and even more recently cloudcomputing, which can provide the basis for a shared infrastructure.However, there are few all-in-one platforms that are particularly suitedfor use in such environments. These are the general areas thatembodiments of the invention are intended to address.

SUMMARY

Systems and methods are provided for providing efficient low-latencymultiplexing (herein referred to as “muxing”) between servers in thecluster. One such system can include a cluster of one or more highperformance computing systems, each including one or more processors anda high performance memory. The cluster communicates over an InfiniBandnetwork. The system can also include a middleware environment, executingon the cluster, which includes one or more application server instances.The system can include one or more selectors, wherein each said selectorcontains a queue of read-ready file descriptors. Furthermore, the systemcan include a shared queue, wherein the read-ready file descriptors ineach said selector can be emptied into the shared queue. Additionally, aplurality of multiplexer (herein referred to as “muxer”) threadsoperates to take work from said shared queue.

Other objects and advantages of the present invention will becomeapparent to those skilled in the art from the following detaileddescription of the various embodiments, when read in light of theaccompanying drawings.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 shows an illustration of a middleware machine environment 100, inaccordance with an embodiment of the invention.

FIG. 2 shows another illustration of a middleware machine platform orenvironment, in accordance with an embodiment of the invention.

FIG. 3 shows an illustration of supporting simple muxing in accordancewith various embodiments of the invention.

FIG. 4 shows an illustration of supporting parallel muxing in accordancewith various embodiments of the invention.

FIG. 5 shows an illustration of supporting work sharing muxing inaccordance with various embodiments of the invention.

FIG. 6 shows an illustration of exemplary pseudo code that supports worksharing muxing in accordance with various embodiments of the invention.

FIG. 7 shows an illustration of other exemplary pseudo code thatsupports work sharing muxing in accordance with various embodiments ofthe invention.

FIG. 8 illustrates an exemplary flow chart for supporting work sharingmuxing in accordance with an embodiment of the invention.

DETAILED DESCRIPTION

Described herein are systems and methods that can support work sharingmuxing in a cluster.

FIG. 1 shows an illustration of a middleware machine environment 100, inaccordance with an embodiment of the invention. As shown in FIG. 1, eachmiddleware machine system 102 includes several middleware machine rackcomponents 104, each of which includes a combination of high-performancemiddleware machine hardware nodes 106 (e.g., 64-bit processors, highperformance large memory, and redundant InfiniBand and Ethernetnetworking), and a middleware machine software environment 108. Theresult is a complete application server environment which can beprovisioned in minutes rather than days or months, and which can scaleon demand. In accordance with an embodiment, each middleware machinesystem can be deployed as a full, half, or quarter rack, or otherconfiguration of rack components, and several middleware machine systemscan be coupled together, again using InfiniBand, to create largerenvironments. Each middleware machine software environment can beprovisioned with several application server or other software instances,for example as shown in FIG. 1, an application server instance 109 couldcomprise a virtual machine 116, operating system 120, virtualizationlayer 124, and application server layer 128 (e.g. WebLogic, includingservlet 132, EJB 134, and Gridlink 136 containers); while anotherapplication server instance 110 could comprise a virtual machine 118,operating system 122, virtualization layer 126, and data grid layer 140(e.g. Coherence, including an active cache 142). Each of the instancescan communicate with one another, and with both its middleware machinehardware node, and other nodes, using a middleware machine integrationcomponent 150, such as an ExaLogic integration pack, which itselfprovides several optimization features, such as support for InfiniBandand other features, as described in further detail below.

FIG. 2 shows another illustration of a middleware machine platform orenvironment, in accordance with an embodiment of the invention. As shownin FIG. 2, each application server instance can act as a sender and/orreceiver 160, 161 within the middleware machine environment. Eachapplication server instance is also associated with a muxer 162, 163,that allows the application servers to communicate with one another viaan InfiniBand network 164. In the example shown in FIG. 2, anapplication server instance can include a kernel space 165, user space167, and application server (e.g. WebLogic space) 166, which in turn canbe associated with a sockets direct protocol 168, a JVM (e.g.JRockit/Hotspot layer) 170, a WLS core 172, a servlet container 174, anda JSP compiler 176. In accordance with other examples, othercombinations of middleware-type software can be included. In accordancewith various embodiments, the machine integration component can providefeatures such as Zero Buffer Copies, Scatter/Gather I/O, T3 Connections,Lazy Deserialization, and GridLink DataSource, to provide the basis for,and improve performance within, the shared infrastructure.

Simple Muxing

FIG. 3 shows an illustration of supporting simple muxing in a middlewaremachine environment. As shown in FIG. 3, a file descriptor cache, or FDcache 301, which is maintained by an operating system (OS) in amiddleware machine environment 300, can contain one or more read-readyfile descriptors, such as sockets (marked with cross). Here, scanning alarge cache and/or maintaining concurrent access to a large cache can beinefficient in the OS.

A poll device, such as a selector 302, which may be exposed via aselector interface, can include a queue of read-ready file descriptors,such as sockets (shown as dots in the list). The selector 302 can beused by one or more muxer threads 305 a-305 c to poll the FD cache 301.A thread 305 a-305 c may be blocked at the selector 302, e.g. whileplacing a Selector.select( ) function call, waiting for a scan on the FDcache 301 to complete. Then, the thread 305 a-305 c can copy read-readyfile descriptors into the selector list 302.

Each muxer thread 305 a-305 c can maintain a thread-local list 303 a-303c. The thread-local list 303 a-303 c includes a list of read-readysockets (shown as dots in the list) that can be processed by the thread305 a-305 c. Since the list 303 a-303 c is thread-local, other threadsmay not be able to help processing that list, even when the otherthreads are idle.

As shown in FIG. 3, the thread-local lists 303 a-303 c have a differentnumber of dots, which represent a different amount of work handed overto individual muxer threads. For example, the muxer thread 305 b mayneed to handle a long list of read-ready sockets, while the muxer thread305 c is idle.

A request manager 304 can be used to handle one or more requests fromdifferent servers in the middleware machine environment 300. The requestmanager 304 is a component with multiple queues, to which the requestsprepared by the muxer threads 305 a-305 c can be added. These queues canbe first-in-first-out (FIFO) queues, or priority queues. Additionally,constraints on the thread counts may be enforced on the various queuesin the request manager 304.

As shown in FIG. 3, only one selector 302 is provided for the multiplemuxer threads 305 a-c. For example, when the muxer thread 305 b obtainsa read-ready socket, it has the work all in its own thread-local list303 b. At the same time, other muxer threads 305 a and 305 b may bewaiting to enter Selector.select( ). Thus, the performance of the muxermay suffer, since the read-ready sockets are processed under asingle-thread model.

Furthermore, once the muxer thread 305 b returns from Selector.select( )another thread, e.g. 305 c, may enter Selector.select( ) again. In sucha case, since the selector 302 has just been emptied, it may likelyblock the muxer thread 305 c. Thus, there may be a situation where mostof the muxer threads are waiting, while one muxer thread is busy.

Thus, in the example, as shown in FIG. 3, there may be contention inaccessing the shared selector 302 and FD cache 301 by the differentmuxer threads 305 a-305 c. Furthermore, the non-sharing of read-readysockets can result in thread starvation, where idle threads may beavailable but unable to start processing read-ready sockets.

Additional information about simple muxing is disclosed in U.S. patentapplication Ser. No. 13/109,871, filed May 17, 2011, entitled “SYSTEMAND METHOD FOR PARALLEL MUXING BETWEEN SERVERS IN A CLUSTER”, whichapplication is hereby incorporated by reference.

Parallel Muxing

FIG. 4 shows an illustration of supporting parallel muxing in amiddleware machine environment. As shown in FIG. 4, the middlewaremachine environment 400 can include a number of selectors 402 a-c, eachof which can be used to poll one or more file descriptor caches, or FDcaches 401 a-c. Additionally, there can be exactly the same number ofworker threads, or muxer threads 405 a-c, in the middleware machineenvironment 400, with each muxer thread 405 a-c maintaining athread-local list 403 a-c.

Using parallel muxing, each selector may be accessed by only one muxerthread. For example, the muxer thread 405 a uses the selector 402 a,while the muxer threads 405 b use the selector 402 b and the muxerthreads 405 c use the selector 402 c. The worker threads 405 a-c pollthe selectors 402 a-c single-threadedly and process read-ready socketssingle-threadedly. The using of individual selectors 402 a-c allows areduction in the arrival rate per selector, and, therefore reduces thecontention on the system resources.

Using parallel muxing, the unevenness of load caused by single selectorcan be resolved. Furthermore, there may still be a need for achievingeven distribution of work among different selectors and muxer threads.

Thus, in the example as shown in FIG. 4, the contention in accessing theselectors 402 a-c and FD caches 401 a-c by the muxer threads 405 a-c canbe reduced. Furthermore, the FD cache scanning efficiency can improve byusing several smaller caches 401 a-c, instead of one large cache 301 asshown in FIG. 3, for comparable sets of FDs. Additionally, the design asshown in FIG. 4 may still suffer from thread starvation because that themuxer threads 405 a-c do not share the read-ready sockets.

Additional information about parallel muxing is disclosed in U.S. patentapplication Ser. No. 13/109,871, filed May 17, 2011, entitled “SYSTEMAND METHOD FOR PARALLEL MUXING BETWEEN SERVERS IN A CLUSTER”, whichapplication is hereby incorporated by reference.

Work Sharing Muxing

FIG. 5 shows an illustration of supporting work sharing muxing inaccordance with various embodiments of the invention. As shown in FIG.5, the middleware machine environment 500 includes a shared queue, suchas a concurrent queue of work 506, into which one or many selectors 502a-c can be emptied. The shared queue 506 can contain both read-readysockets and/or related selectors 502 a-c.

By joining the blocking queues of the selectors 502 a-c into one sharedqueue 506, the queue processing model avoid requiring the individualworker threads 505 a-e to process all read-ready sockets sequentially.The worker threads 505 a-e can be activated to enable concurrentprocessing of read-ready sockets from individual selectors 502 a-c.Thus, the shared queue 506 can improve the concurrent properties of theFD caches 501 a-c maintained by the OS, and the queue processing modeloffers reduced end-to-end latency.

Using this queue processing model, the one or many read-ready socketsreturned from individual selectors 502 a-c, and selectors 502 a-cthemselves can be shared among multiple worker threads, or muxer threads505 a-c. As shown in FIG. 5, the system can include one or more muxerthreads 505 a-c, each of which can obtain one read-ready socket from theshared queue of work 506. Thus, each thread-local list 503 a-503 c mayonly have a length of one. Additionally, one or more muxer threads 505d-e can use the selectors 502 a-b to poll the FD caches 501 a-b.

As long as the shared queue 506 is not empty, the muxer threads 505 a-emay not get suspended in order to achieve high throughput for the muxer,since the queue processing model can avoid having some threads blockedin Selector.select while other threads may have more than one socket toprocess. Thus, this queue processing model can reduce the queue waitingtime of requests that otherwise would be wasted in the thread-locallist.

In accordance with various embodiments of the invention, the number ofmuxer threads (MUXERS) can be less than or equal to the number of theselectors (SELECTORS), or 1<=SELECTORS<=MUXERS. The muxer threads canpotentially be blocked in Selector.select( ) at every selector. Thus,there may be up to the number of SELECTORS muxer threads blocked inSelector.select( ). Once a muxer thread returns from the Selector with alist of read-ready sockets, one or more threads may be ready to takework from the shared queue 506, while some of the muxer threads may bebusy reading socket at the time. The number of the threads that areready to read the read-ready sockets can be up to the number ofMUXERS-SELECTORS, which represents the difference between the number ofMUXERS and the number of SELECTORS.

When a muxer thread 505 a-e is idle, the worker can either be blockedtrying to get read-ready sockets from a selector 502 a-c, or be blockedtrying to get read-ready sockets from the shared blocking queue 506.When one or many read-ready sockets become available, the read-readysockets and their selectors 502 a-c can end up in the shared blockingqueue 506 in the order that guarantees system-wide progress.

In accordance with various embodiments of the invention, every workerthread 505 d-e that returns from the selector can retain one lastread-ready socket. Every worker thread 505 a-c that gets unblocked fromthe shared queue 506 can have a read-ready socket. The worker thread 505a-c can continuingly process these sockets (e.g. read a request), andthen return to get more read-ready sockets from the shared queue 506.Eventually a selector 502 a-c can be taken from the shared queue 506, inwhich case the worker thread 505 a-c can proceed to get more read-readysockets from that selector 502 a-c.

Since the order, in which the read-ready sockets are processed, is basedon the selector 502 a-c, there is a larger opportunity for having moreread-ready sockets in the shared queue 506 using this queue processingmodel. As a result, the read-ready sockets can be obtained from theselector 502 a-c without blocking, and significant response timereductions can be achieved for network-intensive workloads.

Furthermore, the sharing scheme enables the worker threads 505 a-e tocontinuously obtain read-ready sockets from the selectors 502 a-c andprocess them without the need to get suspended or to perform contextswitches. Thus, this queue processing model can achieve a great degreeof concurrency.

FIG. 6 shows an illustration of exemplary pseudo code that supports worksharing muxing in accordance with various embodiments of the invention.As shown in FIG. 6, a Muxer class can be defined (Lines 1-31). A queueof work can be set up and populated with a number of Selectors (Line 4).Furthermore, when the Muxer is running, an infinite loop (Lines 13-30)can be used for processing sockets. For each iteration in the infiniteloop, the Muxer class can determine whether an entry obtained from anArrayList is a Selector or a SelectionKey (Line 18). In the case theentry is a Selector, the Muxer can try to push the content in theSelector to queue of work (Lines 18-25). Otherwise, the Muxer canestablish the readReadyChannel using the SelectionKey (Lines 26-29).

FIG. 7 shows an illustration of other exemplary pseudo code thatsupports work sharing muxing in accordance with various embodiments ofthe invention. As shown in FIG. 7, a different Muxer class can bedefined (Lines 1-31) to be more eager in sharing outstanding work, e.g.if the ArrayList is not empty, the Muxer class can push its contents tothe queue of jobs using the jobs.offerAll function call (Lines 17-21).Thus, within the infinite loop, the Muxer class tries to dump non-emptylist of work on every iteration. Additionally, the Muxer class can avoidconcurrent access to the queue in a manner similar to the thread-locallist.

Furthermore, in order to keep as many muxer threads busy reading socketsas possible, instead of waiting in Selector, the Muxer class tries toadd the selector to the queue as late as possible. The reason is that,if the time from the last poll is longer, then it is more likely thatthe worker thread can return immediately, thus blocking less. Otherwise,if a worker thread enters the Selector.select too soon, the call wouldmore likely get the worker thread blocked, since the selector list wasemptied only a short while ago and the file descriptor cache may nothave enough time to get populated again.

FIG. 8 illustrates an exemplary flow chart for supporting work sharingmuxing in accordance with an embodiment of the invention. As shown inFIG. 8, at step 801, the system can provide one or more selectors,wherein each said selector contains a queue of read-ready filedescriptors. Then, at step 802, the system can provide a shared queue,wherein each said selector is emptied into the shared queue.Furthermore, at step 803, the system allows a plurality of muxerthreads, running on the one or more microprocessors, to take work fromsaid shared queue.

In accordance with various embodiments of the invention, the efficiencyof the system can be achieved based on efficient concurrent selection ofone selector out of many selectors, which enables concurrent selectionfrom multiple small FD caches instead of one large cache. Furthermore,the use of non-blocking sharing of read-ready sockets can eliminatethread starvation. It is beneficial to use a non-blocking bulk addoperation for a concurrent queue with a fixed memory footprint (e.g.,jobs.offerAll( . . . ) as shown in line 20 of FIG. 7). Additionally, thesystem can guarantee timely processing of read-ready sockets and canimprove other progress properties of the system as a whole, by ensuringthe order, in which the selector is shared after the sockets areselected (e.g. the order in which the read-ready sockets are added inconnection with the selector).

Thus, using work sharing muxing, the system can guarantee efficientqueue progressing and allow the sharing of various mutable states, andcan eliminate thread starvation during the concurrent processing ofread-ready sockets.

The present invention may be conveniently implemented using one or moreconventional general purpose or specialized digital computer, computingdevice, machine, or microprocessor, including one or more processors,memory and/or computer readable storage media programmed according tothe teachings of the present disclosure. Appropriate software coding canreadily be prepared by skilled programmers based on the teachings of thepresent disclosure, as will be apparent to those skilled in the softwareart.

In some embodiments, the present invention includes a computer programproduct which is a storage medium or computer readable medium (media)having instructions stored thereon/in which can be used to program acomputer to perform any of the processes of the present invention. Thestorage medium can include, but is not limited to, any type of diskincluding floppy disks, optical discs, DVD, CD-ROMs, microdrive, andmagneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flashmemory devices, magnetic or optical cards, nanosystems (includingmolecular memory ICs), or any type of media or device suitable forstoring instructions and/or data

The foregoing description of the present invention has been provided forthe purposes of illustration and description. It is not intended to beexhaustive or to limit the invention to the precise forms disclosed.Many modifications and variations will be apparent to the practitionerskilled in the art. The embodiments were chosen and described in orderto best explain the principles of the invention and its practicalapplication, thereby enabling others skilled in the art to understandthe invention for various embodiments and with various modificationsthat are suited to the particular use contemplated. It is intended thatthe scope of the invention be defined by the following claims and theirequivalence.

What is claimed is:
 1. A system for supporting work sharing multiplexingin a cluster, comprising: one or more microprocessors; a plurality offile descriptor caches that each contain one or more read-ready filedescriptors; a plurality of selectors and a plurality of selectorqueues, wherein each of said plurality of selectors is associated with adifferent selector queue of said plurality of selector queues, andwherein each of said plurality of selectors polls an associated filedescriptor cache of said plurality of file descriptor caches, and writesread-ready file descriptors from the associated file descriptor cache tothe associated selector queue of said plurality of selector queues; asingle shared queue coupled to said plurality of selectors, wherein eachselector queue of said plurality of selector queues empties filedescriptors from the associated selector queue of the plurality ofselector queues into the single shared queue such that all read-readyfile descriptors polled from the plurality of file descriptor caches bythe plurality of selectors are provided to the single shared queue; anda plurality of worker threads, running on the one or moremicroprocessors, that operates to process said read-ready filedescriptors provided to said single shared queue.
 2. The systemaccording to claim 1, wherein said single shared queue includesread-ready file descriptors and related selectors of said plurality ofselectors.
 3. The system according to claim 2, wherein: a threadassociated with a said selector is blocked when it is waiting for a scanon the associated file descriptor cache to complete.
 4. The systemaccording to claim 2, wherein: a thread associated with a said selectoroperates to copy the one or more read-ready file descriptors into theone or more selectors.
 5. The system according to claim 1, wherein: eachsaid worker thread maintains a thread-local list of read-ready socketsthat it is processing.
 6. The system according to claim 1, furthercomprising: a request manager that is capable of handling one or morerequests from different servers in the cluster.
 7. The system accordingto claim 6, wherein: the request manager contains multiple queues wherethe one or more requests prepared by the plurality of worker threads areadded to.
 8. The system according to claim 1, wherein: there are moresaid worker threads than said selectors.
 9. The system according toclaim 1, wherein: said worker threads operate to continuously obtainread-ready sockets from the shared queue and process them without theneed to get suspended or to perform context switches.
 10. The systemaccording to claim 1, wherein: a worker thread that returns from a saidselector can retain one last read-ready socket.
 11. A method forsupporting work sharing multiplexing in a cluster, comprising: providinga plurality of file descriptor caches that each contain one or moreread-ready file descriptors; providing a plurality of selectors and aplurality of selector queues, wherein each of said plurality ofselectors is associated with an associated file descriptor cache of theplurality of file descriptor caches and a different selector queue ofsaid plurality of selector queues; providing a single shared queuecoupled to said plurality of selectors; polling said plurality of filedescriptor caches with said plurality of selectors wherein each selectorpolls the associated file descriptor cache; writing read-ready filedescriptors from each selector of said plurality of selectors to theselector queue associated with said each selector; emptying theread-ready file descriptors from all of the plurality of selector queuesinto the single shared queue such that all read-ready file descriptorspolled from the plurality of file descriptor caches by the plurality ofselectors are provided to the single shared queue; and processing saidread-ready file descriptors provided to said single shared queue with aplurality of worker threads.
 12. The method according to claim 11,wherein said single shared queue includes read-ready file descriptorsand related selectors of said plurality of selectors.
 13. The methodaccording to claim 12, further comprising: configuring a threadassociated with each selector to be blocked when it is waiting for ascan on the associated file descriptor cache to complete.
 14. The methodaccording to claim 12, further comprising: configuring a threadassociated with each selector to copy the one or more read-ready filedescriptors into the one or more selectors.
 15. The method according toclaim 11, further comprising: configuring each said worker thread tomaintain a thread-local list of read-ready sockets that it isprocessing.
 16. The method according to claim 11, further comprising:providing a request manager that is capable of handling one or morerequests from different servers in the cluster.
 17. The method accordingto claim 16, further comprising: configuring the request manager tocontain multiple queues where the one or more requests prepared by theplurality of worker threads are added to.
 18. The method according toclaim 11, further comprising: providing more worker threads than saidselectors.
 19. The method according to claim 11, further comprising:configuring one or more said worker threads to continuously obtainread-ready sockets from the selectors and process them without the needto get suspended or to perform context switches.
 20. A non-transitorymachine readable storage medium having instructions stored thereon forsupporting work sharing multiplexing in a cluster, which instructions,when executed, cause a system to perform steps comprising: providing aplurality of file descriptor caches each containing one or moreread-ready file descriptors; providing a plurality of selectors and aplurality of selector queues, wherein each of said plurality ofselectors is associated with an associated file descriptor cache of theplurality of file descriptor caches and a different selector queue ofsaid plurality of selector queues; providing a single shared queuecoupled to said plurality of selectors; polling said plurality of filedescriptor caches with said plurality of selectors wherein each selectorpolls the associated file descriptor cache; writing read-ready filedescriptors from each selector of said plurality of selectors to theselector queue associated with said each selector; emptying theread-ready file descriptors from all of the plurality of selector queuesinto the single shared queue such that all read-ready file descriptorspolled from the plurality of file descriptor caches by the plurality ofselectors are provided to the single shared queue; and processing worksaid read-ready file descriptors provided to said single shared queuewith a plurality of worker threads.